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ABSTRACT 

Described here is a rating scale to provide feedback 
about the success of training and of program implementation when a 
new program is biing adopted. It allows pro ject personnel to 
determine whether trainees possess the knowledge and skills necessary 
to implement a new program for which they are being trained. It also 
informs project personnel of the extent to which a program has been 
implemented. The elements of the rating scale include columns 
containing descriptions of the activities and attributes associated 
with the educational program, criterion levels for each attribute, 
data sources and monitoring or documenting activities for measurement 
of each attribute, reports of the level of implementation, and an 
indication of whether the criterion level has been met. The level of 
implementation is stated in terms of precise levels of compliance. 
Information concerning the implementation of program attributes is 
collected by internal and external prcgram evaluat.ors using multiple 
techniques. Programs adapted for use in new situations can be 
evaluated using the same attribu* »nd criteria used for the 
original program. (Author/JM) 
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^ ^yJ^Jlgj^'0_r^Q!L an . c ^ n g Training and Monitoring of 
j^l'jgram Adapt lot i_s and Adoptions in N ew S i t ua 1 1 o n s 

(i.iry Stuck and Roberta Rubin 
University of North Carolina 

The dissemination and implementation of innovative educational 
prolans are of vital interest in education, Organizations Mich as the 
National Diffusion Network (M>N) function to disseminate validated 
programs to interested and new I ! Vs nationwide, ihc assumption under- 
l>mg the validation of projects is that -uch pioiects can be transported 
ard can be implemented in much the same way as that in which *"hoy were 
original Lv validated. Ixpeiuncc has demonstrated, however, that LFAs 
often cn:nunti:r substantial difficult when t ht*y attempt to implement the 
nnoiatiw, validated proji-Lt. It i,s ipparuit that the Miccessiul imple- 
mentation of an innovative pi agram m a new situation is dependent on 
the training local personnel receive and the provision of formative 
evaluation. 

Implementation may be viewed as an attempt to put into practice an 
idea, program, or set of activities which is new to the individual or 
organization using it. Various investigators have studied the educational 
change and implementation process. Their conclusion is that successful 
imp lenient at ion depends, in p.»rt, on circumstances specific to the local 
situation and the strategies used to insure installation in the new 
situation, laillan and Pomfret f 1 977 ) identify several mechanisms 
important for implement it ion: (1) training; (2) resource support; (3) 
feedback mechanisms; and i\) participation in decision-making 

Most educational programs which have been transported and implemented 
in adapting or adopting communities have not been evaluated for the level 
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of implementation of their innovation, tvaluati ; the extent of imple- 
mentation of a program permits piogram developers to define critical 
attributes of the program. Once these critical attributes na\o Ken 
defined, the criteria for judging an adaption of adoption can be established 
Most programs have not specified these criteria and consequently, the 
ms,]y implemented adapting or adopting program do not know the extent 
rf implementation that required for an adaption as compired to an 
adoption. lerv often, adaptions and adoptions are indistinguishable from 
one .mother In Jdition, the new adaptirg or 'adapting site; do not know 
the relevance of \ | K attribute 1 - being implemented. One u\i*on for this 
lack of implementation « nfoim.it i on collected bv riograms for their "new" 
adapting or adopting programs 1, the concern about the imposition placed 
upon these new programs once implementation data collection i, requ * ed . 
'Ire "original" urograms may fear losing potential adaptions or adoptions 
once verification procedures are imposed upon "new" LL'Ns. 

It is necessary that staff in "new" communities be subjected to 
nctnoJs of verification that may be used to injure whether th»-y have 
acquired the skills and knowledge necessar\ to implement the program. 
Therefore, service and inservice training of the key personnel m a 
pro-ram n essential for its successful implementation. Although some 
kind of training for tho program par t i c i punts is usually prescribed and 
pruct'eed, ury often a competency or achicviment test is used is evidence 
ot Micco-.sful cofplition of the iniinniv:. Vh know frcm experience that 
then* is a problem in translating what is irarneel in the classroom to a 
real- J fc situation In norio cases, prog i am developers have n»'t prepared 
for the unexpected and unanticipated vagaries f»f the local situation, 
\iso, when a program is initially implemented, the provision of appro- 
priate time and material resources may prove inadequate . 



\n appropriate evaluative and feedback mechanism which stimulates 
interaction and problem identification among the program developers and 
molementers i -> essential for successful implementation, There a 
need to collect information which may identif) difficulties encountered 
during implementation Once problems arc dentil icd, attempts can then 
be made to alleviate them. The program developers and program imple- 
ments should participate in the process of finding solutions to the 
identified implementation problems. T n this way, viable solutions to 
th'- pioMen^ of implementation jimv be found which fit the local situation, 
lb* 1 present mthor., became particular!} aware of tbr pi obi cms asso- 
-J lv 'th ! aipl n t in : piue.t.i'ns ..r.d pnetur <i i result of their own 
i:m>ht. -ne.it :n uveral Mich efforts i hey ^orK 1 u Jed , independently, that 
th" *>rohknis occurred at tw<< Uages, the training stage and the implc- 
RMPt.:tion s'ta^e, iurther, lhe\- concluded that full implementation was 
dependent upon the intercut and the requisite competencies required to 
implement fully, Varia'ions among "adopt ions" can usually be attributed 
to these two factors, intent and technical competency. The procedure 
discussed in this paper for enhancing training and monitoring of program 
adaptions and adoptions relates to the technical competencies required 
for the proper installation of a product or practice that has been trans- 
ported, 

I rom the technical standpoint, proper training for and implementation 
of a program depends upon knowing what are the critical attributes of the 
program being implemented, The implementer should also know the criterion 
level that constitutes implementation. Too often, programs arc described 
in general terms and training is based upon these general descriptions, 
Sr Idom dc\s the person being trained know exactly which program attributes' 
arc "absolutely essential" and which ones are "desirable. " fcven when 



programs are imp k riK nt ed by well trained liuI l v uluals , program changes 
uttui occur becae^e of turnover in personnel. A svstem that ensures 
quality control and maintenance usually does not exist. The method 
described in this paper is one that should be useful in ongoing training 
and monitoring activities to ensure the maintenance of proper program 
implementation. 

Any effective training and monitoring system must provide feedback 
to trainees and project personnel. In the case of training, the feed- 
back should allow r lie trainee and the project personnel to accurately 
determine when he/she possesses the knowledge and skills necessary to 
implement a given program. Likewise, in the ca ;c o r Implementation, 
the feedback should inform the imp lom^nter of the extent to winch a 
prn*;r\im h. - ncen implemented. Obvious]}, this reuures more than indi- 
Ctitiiu; to the i mploment ei whether or \ ot a particular facet >f the 
program h.'s been implemented. Rather, the feedback shoild inform the 
: npl ementer of the extent to which a criterion level, indicative of 
pro;.er inplenvntat ion, has been reached. The need for implementation 
ieedback never ends. Not only should implementation data be collected 
and feedback provided, the implementat i on data hould also be related 
to achievement outcomes. Thi » is the only uay it is possible to establish 
over time in a given situation which program attributes are responsible 
for program effects. 

Before work was begun on the training and monitoring method described 
in the following paragraphs, a search of existing tools was conducted. 
The search revealed that various strategics exist tor assessing the degree 
of implementation of innovative curriculum program- (Alkin, 1909; Hail 6 
Loucks, 1977; Leinha^dt, 1»76; Leithwood 6 Montgomery, 1980; Stallings, 
l f )7T»J Most of then>, however, possess serious tortcomings for determining 



the implementation of programs which are not classroom-oriented (Revicki, 
Rubin, 5 Stuck, 1981), Therefore, work was begun on the training and 
monitoring method discussed next in this paper. 

The Proposed Method 
The proposed method involves the development and use of a rating 
scale. The elements of the rating scale include columns which contain 
descriptions of the activities and attributes associated with the educa- 
tional program, criterion level for each attribute, data sources and 
monitoring/documenting activities for measurement 01 .ach attribute, 
reports of the level of implementation, and an indication of whether the 
criterion level has been met. for each attrtbute, a criterion level is 
specified and the monitoring activity associated with the attribute is 
indicated. There is also a specification of the level of implementation 
of each program feature. This is not stated in ter^s of compliance 
versus noncompliance but rather, in precise levels of compliance. Finally, 
an indication of whether the criterion level was achieved is recorded on 
this rating scale (Rubin, Stuck, f, Pevicki, 1982), 

The first step in developing the rating scale is to identify the 
major a ttr i bute s of the particular education program (Morris 6 Fitz- 
Gibbons, 1978). The set of attributes will var> across different programs. 
The range of activities which constitute these programs should be specified 
by the model developer. Only those persons from the model developer's 
shop who are incimately familiar with the operation should participate in 
selecting the essentia 1 elements or attributes of the program. If the 
program is going to be transported, a minimum set of these attributes for 
implementation should be specified at the outset by the model developer. 
This minimum set of core components of the program will help to facilitacr 
the training and implementation processes. 
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After the major program attributes and characteristics have been 
agreed upon, the specification of implementation levels is required for 
each attribute. For each attribute, a criterion level should be speci- 
fied and the associated data collected and reported as precisely as 
possible. The results should indicate the extent to which an objective 
was or was not achieved. The information about the level of achievement 
should be reported as a continuous variable, whenever possible (Rubin, 
Stuck, f, Revicki, in pressj^ In this paper, several examples of attri- 
butes from two very different types of program;, will be described. The 
first program involves a learning for masteiy strategy which was 
developed by Bloom (1968) and was based upon the work of Carroll (1963). 
The second program, the Parent Education Follow Through Program (POFTPJ , 
is a field-based parent education program within the national Follow 
Through Project. 

Two examples of attribute:-, from the Learning for Mastery approach 
are shown in Table 1. As noted in this table, the following columns are 
included in the rating scale: descriptions of the activities and attri- 
butes associated with the innovation; criterion levels for each attribute; 
data sources and monitoring/documenting activities for measurement of < ach 
attribute; reports of the level of implementation; and an indication of 
whether the criterion level has been met. 

Attribute number 1 addresses the identification of items on a progress 
test that were answered incorrectl> ->y nonmasters. If at least 5, or 50%, 
o r the nonmasters missed a particu. item, then group correctives should 
be used. The criterion le vel for analyzing the items correctly in order 
to assign group or individual correctives is 90%, As seen in the example 
in Table 1, under the column labelled monitoring/document a tion activity 
and data source , the external evaluator's results should be compared to 
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tin. teacher's results. In this example, 95% of the formative te^t items 
were analyzed correctly and the criterion level was met. Similarly, the 
criterion level for attribute number 2 was met (see Table 1). In this 
case, the appropriate kind of corrective (e.g., group or individual) 
should be administered for 80" of the l terns analyzed. The monitoring 
activity needed to document this attribute Involves the observation of 
instruction to determine the correspondence between the appropriate type 
of corrective previously assigned and the corrective actually administered. 

In contrast to the Learning for Mastery approach in which implementa- 
tion takes place in a school environment, the Parent Education Follow 
Through Program (PEFTPJ is a fie'd-based educational program which is 
primarily implemented in the homo and the school. The rating scale 
referred to in this paper can be easily used as a training and implemen- 
tation device in both of these types of programs. 

Presented in Table 2 are two examples of activities associated with 
the PKI-TP. Attribute number 1 is related to home visitation frequencies 
by the paraprof cssiona Is employed in this program. The criterion level 
associated with this attribute is specified as having 50% of the para- 
professional 1 s employment time spent in making home visits and having at 
least 3/4 of the scheduled home visits eompletd for at least 80% of the 
program children. The monitoring/documentin g activity associated with 
this attribute involves checking the reported percentage of time spent 
in home visits which is included on a paraprofessi onal 1 s weekly report. 
The monitoring/documenting activity also involves observing a random 
sample of paraprofessionals in a subsample of their visits for a week. 
It should be noted that the other attribute stated in this table follows 
the same pattern. For example, attribute number 2, attendance at Policy 
Advisory Committee meetings, has a criterion level set at 35% attendance 
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and has two types of monitoring activities (e.g., record review and 
observation) (sociable 1). 

These examples serve to emphasize the point that the attributes of 
programs are as diverse as having a very structured classroom situation 
to a less structured home environment situation. The proposed method 
can be used with all types of programs, 

Adva ntages of the Proposed Method 

Preservice and inservice training is required for the effective 
implementation of any innovative educational program. The goals and 
objectives of the training program must be consistent with the attributes 
and criterion levels specified in the rating scale. These, of course, 
arc derived from the program being implemented. If this is the case, the 
rating scale may be used as a measure of the effectiveness of the training 
in bringing about observable changes in the behaviors of the program 
participants consistent with the design of the innovative program. Dis- 
crepancies bctwoe* the expected and observed behaviors are indicative of 
less than successful transferral of program objectives into practice in 
the local situation, further in-service training may be required to 
alleviate the observed lack of program implementation. 

The rating scale may also be used to monitor and provide feedback 
regarding the level of implementation in the local situation. It may also 
provide valuable information related to the operation of the various 
program features. Implementation evaluation data could he collected 
periodically to keep program administrators informed and to identify 
problems in the implementation process, Systematic feedback could be 
provided to the program personnel concerning their behavior and its 
approximation to the implementation objectives of the program. Problems 
that are identified may be solved jointly through the cooperation of 



program developers and personnel in the field. This ensures the partici- 
pation of all relevant stakeholders and improves the likelihood of program 
survival . 

hducat i onal Implica tions * 
Programs implemented in new situations should be evaluated using the 
same attributes and criteria that are used for the "original'* program from 
which it is adapting or adopting. This evaluation would help to insure the 
fidelity 0 f the program in "new" locations. In addition, program developers 
can gain further understanding of the operation of certain attributes of 
their program. Clearly, the end result would be a more successful program 4 
adapted to the LCA's needs. Research could be Conducted which would Ink 
the program attributes to desired outcomes which would contribute to the 
identification of the program features most related to the outcomes, Once 
these prouram features have been defined., program developers may then modify 
their program resulting., in -j more cost effective and improved educational 
program which is responsive to the needs of the constituents that it serves. 
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Table I 

Sdinpio Training and implementation Rating Scale' 
Learning for Mastery Approach 



Activity or Attribute 
Associated with the Program 



1, identifying items on diagnostic 
progress tesc that many (e.g., 
£ 501 cr 1 5) and few 
(e.g. , < 501 or £ 4) of the 
nonmasters answered incorrectly. 



Criterion Level 



Mom t or ing/ Document i ng 
Activity and Data Source 



f ,evel of 

Imp lementat ion 



Criterion Level 
Met or Not Met 



90% ot the formative test items 
will be analv^eu correctly by 
the teacher 



Analyze formative test results 
to identify students for group 
or ind iv ldua \ correc t ives 

Compare external evaluator' results 
with the teachet's result** regarding 
group or individual correctives 



95; 



Yes 



Providing group correctives 
related to items missed by many 
(e.g. 50* or! 5) of the non- 
masters and individualized 
correctives related to items 
missed by few (e.g.,<50* or 
4) of the nonmasters. 



Adapted from: 



"or 80* of the items, the 
appropriate kind of corrective 
(e.g., group vs. individual) 
will be administered 



Observe instruction to dettrrine the 
correspondence between the appropriate 
type of corrective determined b.y the 
external evaluation uf items uussed 
by nonmasters and the correctives 
administered. 



Yes 



Rubin, R.I., Stt C. , & Revicki, U.A. A model for assessing the degree of 
implementation in field based education programs. Education Evaluation and 
Policy Analysi s . in press. 




Table 2 

Sample Training and Implementation Rating Scale: 
Parent Education Follow Through Program 



Activity or Attribute 
Associated with the Program 



Cri ter ion 
Level 



Visiting homes of Program 
children by paraprof ess ionals 



50? of paraprofess ional 1 s 
employment time 

At least 7521 of the scheduled 
home visits will be completed 
for at least 80% of the program 
children 



Moni toring/Document ing 
Activity and Data Source 



Level of 
Implementation 



Criterion Level 
Met or Not Met 



Check for the repor ted percentage 
of time in the paraprof ess lona 1 1 s 
weekly reports of the randomly selected 
parap r of e ss ionals . Observe para- 
professionals in a subsample of 
their visits for a week* Do this 
tor ?l least 5 randomly selected 
paraprofessionals . 



$02 Home Visits 
50% Ciassroon 



75% 



Ves 



Yes 



2. Attending Policy Advisory 
Committee (PAC) meetings 
by parents 



At least 35% of the parents 
attend one PAsC meeting during 
the school year 



Check for randomly selected sign-in 
PAC a t tendance shee ts for those 
parents affiliated with the randomly 
selec ted paraprofessionals . 

Observe a PAC meeting and vex if y the 
sign-in sheets at the meeting. 



40% 



Yes 



Adapted from: Rubin, Stuck, C, , & Revicki, D.A. A model for assessing the degree of 

impl^aantafcoon in field based education programs. . Education Evaluation and 
Policy AnalysS aV in press. ~~~* " 



